Recent advances in multi-agent reinforcement learning (MARL) have created opportunities to solve complex real-world tasks. Cybersecurity is a notable application area, where defending networks against sophisticated adversaries remains a challenging task typically performed by teams of security operators. In this work, we explore novel MARL strategies for building autonomous cyber network defenses that address challenges such as large policy spaces, partial observability, and stealthy, deceptive adversarial strategies. To facilitate efficient and generalized learning, we propose a hierarchical Proximal Policy Optimization (PPO) architecture that decomposes the cyber defense task into specific sub-tasks like network investigation and host recovery. Our approach involves training sub-policies for each sub-task using PPO enhanced with domain expertise. These sub-policies are then leveraged by a master defense policy that coordinates their selection to solve complex network defense tasks. Furthermore, the sub-policies can be fine-tuned and transferred with minimal cost to defend against shifts in adversarial behavior or changes in network settings. We conduct extensive experiments using CybORG Cage 4, the state-of-the-art MARL environment for cyber defense. Comparisons with multiple baselines across different adversaries show that our hierarchical learning approach achieves top performance in terms of convergence speed, episodic return, and several interpretable metrics relevant to cybersecurity, including the fraction of clean machines on the network, precision, and false positives on recoveries.

Moore's Law, advocated by Gordon Moore of Intel fame, says that the computational capabilities will double every 18 to 24 months. And we've seen that really unfolding over the last 30 years (see chart). It's really stoked people's imagination, so much so that many believe that the promise of artificial intelligence (AI) could become reality, and computers could actually learn to think like humans. I believe it's still a number of years away, but it is fueling a lot of hype regarding AI. What it's truly capable of, where it can be effective, and what it takes to implement it, all of which have become somewhat inflated in the market today.

There has been a lot of hype around AI to the point where some people are simply tuning it out. I think this is a mistake. While there are limits to what AI can do, there also are sophisticated attacks that we'd miss without it. The need for AI is driven by three fundamental yet significant changes in the enterprise computing environment. Taking all of these factors together leads me to believe that AI is not only a viable solution, but it may be the only solution.

In future autonomous systems, wireless multi-hop communication is key to enable collaboration among distributed agents at low cost and high flexibility. When many agents need to transmit information over the same wireless network, communication becomes a shared and contested resource. Event-triggered and self-triggered control account for this by transmitting data only when needed, enabling significant energy savings. However, a solution that brings those benefits to multi-hop networks and can reallocate freed up bandwidth to additional agents or data sources is still missing. To fill this gap, we propose control-guided communication, a novel co-design approach for distributed self-triggered control over wireless multi-hop networks. The control system informs the communication system of its transmission demands ahead of time, and the communication system allocates resources accordingly. Experiments on a cyber-physical testbed show that multiple cart-poles can be synchronized over wireless, while serving other traffic when resources are available, or saving energy. These experiments are the first to demonstrate and evaluate distributed self-triggered control over low-power multi-hop wireless networks at update rates of tens of milliseconds.